Visual Semantics - Or: What You See is What You Compute

We introduce visual graphs as an intermediate representation between concrete visual syntax and abstract graph syntax. In a visual graph some nodes are shown as geometric figures, and some edges are represented by geometric relationships between these figures. By carefully designing visual graphs and corresponding mappings to abstract syntax graphs, semantics definitions can, at least partially, employ a visual notation while still based on abstract syntax. Visual semantics thus offers the " best of both worlds " by integrating abstract syntax and visual notation. These concepts can also be used to give visual semantics for traditional textual formalisms. As an example we provide a visual definition of Turing machines. Language semantics are conveniently expressed based on abstract syntax. This allows to abstract from details of concrete syntax and leads to more succinct semantics definitions , and in some cases it makes semantics definitions even tractable at all. In [5] we have presented a framework for defining semantics of visual languages. The approach is fundamentally based on an abstract graph syntax for visual languages. Thus, it is essentially a textual formalism which is unfortunate (to a certain degree) for two reasons: first, visual relationships, such as inside or adjacent, are represented in a topological way by appropriately labeled edges. By this transition from a visual to a textual representation , many characteristics of the visual language under consideration are difficult to grasp in the abstract representation, they might even get lost. Second, the tex-tual treatment of a visual language presents in a sense a " modality mismatch " , more precisely, it means a retrogres-sion from visual to textual. This might cause psychologically grounded reluctances to using the formalism – just because it is not visual. These drawbacks might be obstacles for a widespread use of the formalism in the VL community. Therefore, we extend the semantics formalism to stay, to a large degree, with visual notation when defining semantics. The main idea is to re-visualize some relationships of abstract syntax that have been translated more or less directly from geometric relationships. This means that some nodes are visualized as geometric figures, and some of the edges between these nodes are represented by relationships that hold between the figures. Since this will not cover, in general, all relationships, we arrive at a semi-visual notation, that is, a mixture of graphs and pictures, called visual graphs, in which complex relationships are still …